Method for providing virtual image to user in head-mounted display device, machine-readable storage medium, and head-mounted display device

ABSTRACT

A method is disclosed for providing a virtual image to a user in a Head-Mounted Display (HMD) device. The method includes detecting an ambient illumination, calculating a target transmissivity of window provided in the HMD device, based on the ambient illumination, adjusting a transmissivity of the window based on the calculated target transmissivity, and providing a virtual image to the user by projecting light to the window from a projector provided in the HMD device.

PRIORITY

This application claims priority under 35 U.S.C. §119(a) to KoreanPatent Application Serial No. 10-2012-0118251, which was filed in theKorean Intellectual Property Office on Oct. 24, 2012, the contents ofwhich are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to a wearable display device,and more particularly, to a Head-Mounted Display (HMD) device.

2. Description of the Related Art

A conventional, see-through HMD device includes one or two cameras foracquiring an image of an actual surrounding environment. The videosee-through HMD device synthesizes an actual image input from a camerawith a virtual image generated by a computer by using a videosynthesizer, and presents the synthesized image to a user through adisplay such as a Liquid Crystal Display (LCD) attached on the HMDdevice.

The video see-through HMD device may accurately display a virtual objectin a desired position by using a computer vision technique. However,since the display screen is small, the resolutions of both actual andvirtual environments are limited.

SUMMARY OF THE INVENTION

The present invention has been made to address the problems and/ordisadvantages described above.

Accordingly, various aspects of the present invention provide an HMDdevice that simultaneously provides a virtual image and an actual image,reduces the weight and size of the HMD device, improves the outdoorvisibility of the virtual image, and has low power consumption and lowheat emission.

Other aspects to be provided in the present invention may be understoodby embodiments described below.

According to an aspect of the present invention, there is provided amethod for providing a virtual image to a user in an HMD device,including detecting an ambient illumination, calculating a targettransmissivity of a window provided in the HMD device, based on theambient illumination, adjusting a transmissivity of the window based onthe calculated target transmissivity, and providing a virtual image tothe user by projecting light to the window from a projector provided inthe HMD device.

According to another aspect of the present invention, there are provideda computer-readable storage medium having recorded thereon a program forexecuting a method for providing a virtual image to a user in an HMDdevice, and an HMD device including the storage medium.

According to another aspect of the present invention, there is providedan HMD device for providing a virtual image to a user, including asensor unit configured to detect an ambient illumination, a projectorconfigured to project light, a window configured to condense and reflectthe light projected by the projector and to provide a virtual imageformed by the reflected light to the user, and a controller configuredto calculate a target transmissivity of the window corresponding to thedetected ambient illumination and adjust a transmissivity of the windowaccording to the target transmissivity.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of certainembodiments of the present invention will be more apparent from thefollowing detailed description taken in conjunction with theaccompanying drawings, in which:

FIG. 1 illustrates an HMD device viewed from an outer side according toan embodiment of the present invention;

FIG. 2 illustrates an HMD device viewed from an inner side according toan embodiment of the present invention;

FIG. 3 illustrates a structure of an HMD circuit according to anembodiment of the present invention;

FIG. 4 illustrates a schematic structure of a first projector accordingto an embodiment of the present invention;

FIG. 5 illustrates a structure and a function of a first windowaccording to an embodiment of the present invention;

FIG. 6 illustrates a structure of first glass according to an embodimentof the present invention;

FIGS. 7A and 7B illustrate holographic patterns of first HolographicOptical Elements (HOEs) according to an embodiment of the presentinvention;

FIGS. 8A through 9B are views describing transmissivity control of anHMD device according to an embodiment of the present invention;

FIG. 10 illustrates a method for providing a virtual image according toan embodiment of the present invention;

FIGS. 11 through 13 are views describing a first example of a method forproviding a virtual image according to an embodiment of the presentinvention;

FIG. 14 illustrates a second example of a method for providing a virtualimage according to an embodiment of the present invention;

FIG. 15 illustrates a third example of a method for providing a virtualimage according to an embodiment of the present invention;

FIG. 16 illustrates a fourth example of a method for providing a virtualimage according to an embodiment of the present invention; and

FIG. 17 illustrates a fifth example of a method for providing a virtualimage according to an embodiment of the present invention.

Throughout the drawings, like reference numerals will be understood torefer to like parts, components, and structures.

DETAILED DESCRIPTION OF THE EMBODIMENTS OF THE INVENTION

Embodiments of the present invention will be described with reference tothe accompanying drawings. Throughout the drawings, like referencenumerals will be understood to refer to like parts, components, andstructures. In the following description of the present invention, adetailed description of known functions and configurations incorporatedherein will be omitted for the sake of clarity and conciseness.

Although ordinal numbers such as “first,” “second,” and so forth will beused to describe various components, those components are not limited bythe terms. The ordinal terms are used only for distinguishing onecomponent from another component. A first component may be referred toas a second component and likewise, a second component may also bereferred to as a first component, without departing from the teaching ofthe present invention. The term “and/or” used herein includes any andall combinations of one or more of the associated listed items.

When it is mentioned that a component is “connected to” or “accessed by”another component, it may be understood that the component is directlyconnected to or accessed by the other component or that still anothercomponent is interposed between the two components. When it is mentionedthat a component is “directly connected to” or “directly accessed by” toanother component, it may be understood that no component is interposedtherebetween.

FIG. 1 illustrates an HMD device viewed from an outer side according toan embodiment of the present invention, and FIG. 2 illustrates an HMDdevice viewed from an inner side according to an embodiment of thepresent invention.

An HMD device 100 has the appearance of glasses or eyewear, and is aportable terminal (or a portable communication terminal). The HMD device100 includes an HMD housing 110 and an HMD circuit (shown in FIG. 3)mounted in the HMD housing 110.

The HMD housing 110 includes a front frame 115 in which a first window280 and a second window 285 corresponding to the left eye and the righteye are respectively fixed, and a first temple frame 120 and a secondtemple frame 121 that are inwardly folded or outwardly unfolded by afirst hinge 122 and a second hinge 123. Hereinafter, the left eye andthe right eye are referred to as a first eye and a second eye,respectively. In the present invention, the first window 280 and thesecond window 285 ca be referred to as a first window panel and a secondwindow panel, respectively.

A camera 260 is disposed on an outer surface of the front frame 115, andthe camera 260 is disposed in a portion of the front frame 115 betweenthe first window 280 and the second window 285 (that is, in a bridgearea of conventional glasses).

A microphone 212 is disposed on the outer surface of the front frame115, and a touch sensor 250 is also disposed on the outer surface of thefront frame 115.

In an inner surface of the front frame 115, a first opening 111 isdisposed and provides a path through which first projection light 310output from a first projector (shown in FIG. 3) disposed in the frontframe 115 is output to the outside of the front frame 115. A secondopening 112 is disposed in the inner surface of the front frame 115, andprovides a path through which second projection light 320 output from asecond projector (shown in FIG. 3) disposed in the front frame 115 isoutput outside the front frame 115. Transparent protection glass forblocking introduction of external dust into the front frame 115 may beinstalled in each of the first opening 111 and the second opening 112.

At least one button 213 is disposed on the outer surface of the firsttemple frame 120.

At least one speaker 211 is disposed on the inner surface of the firsttemple frame 120 and/or the second temple frame 121.

The first projector outputs the first projection light 310 for forming afirst virtual image, and the first projection light 310 output from thefirst projector is condensed and reflected by the first window 280. Thecondensed and reflected first projection light 312 forms the firstvirtual image on the retina of the left eye 330 of the user. Condensingincludes convergence of light into one point or reduction of a beam spotsize of light. Preferably, the reflected first projection light 312converges to the crystalline lens or pupil of the left eye 330.

The second projector outputs the second projection light 320 for forminga second virtual image, and the second projection light 320 output fromthe second projector 275 is condensed and reflected by the second window285. The condensed and reflected second projection light 322 forms thesecond virtual image on a retina of the right eye 335 of the user.Preferably, the condensed and reflected second projection light 322converges to the crystalline lens or pupil of the right eye 335.Although two projectors are used in the current example, only oneprojector may also be used.

Although the first virtual image and the second virtual image aresimilar to each other except that they are displayed on the left eye andthe right eye, the present invention is not limited thereto, and one ofthe first virtual image and the second virtual image may be displayed.

FIG. 3 illustrates a structure of the HMD circuit 101 according to anembodiment of the present invention.

The HMD circuit 101 includes an input/output module 210, a memory 220, asensor unit 225, a battery 230, a power manager 235, a communicationunit 240, the touch sensor 250, a camera 260, a first projector 270 anda second projector 275, a first window 280 and a second window 285, anda controller 290.

The input/output module 210 is a device for receiving a user input,providing information to the user, receiving data from an externalsource, or outputting data to an external device, and includes at leastone speaker 211, at least one microphone (MIC) 212, at least one button213, a connector, a keypad, or a combination thereof.

A speaker 211 outputs sound corresponding to various data (e.g.,wireless data, broadcast data, a digital audio file, a digital videofile, and a picture) to the outside of the HMD device 100 under controlof the controller 290, or sound corresponding to a function executed bythe HMD device 100. One or more speakers 211 are formed in a properposition of the HMD housing 110. In FIGS. 1-3, two speakers 211 aredisposed on an end portion of the first temple frame 120 and an endportion of the second temple frame 121, respectively.

The microphone 212 receives voice or sound from outside the HMD device100, generates an electric signal, and outputs the generated electricsignal to the controller 290. One or more microphones 212 may be formedin a proper position or proper positions of the HMD housing 110. In thecurrent example, one microphone 212 is disposed on the outer surface ofthe front frame 115. Herein, the term “signal” may also be referred toas “data” and the term “data” may also be referred to as a “datasignal”.

The button 213 is provided to receive a user input, and is used to turnon/off the HMD circuit 101 or select and/or search for a menu item or anicon. The button 213 includes a power button, a volume button, a menubutton, a home button, a back button, navigation buttons (a left button,a right button, an up button, and a down button), or a combinationthereof One or more buttons 213 may be formed in a proper position ofthe HMD housing 110. In FIGS. 1-3, the button 213 is disposed on theouter surface of the first temple frame 120.

A connector may be used as an interface for connecting the HMD device100 with an external electronic device or a power source (notillustrated). The connector is connected with a connector of theelectronic device directly or through a wired cable, and through suchconnector connection, the controller 290 transmits data stored in thememory 220 to the electronic device or receives data from the electronicdevice. The HMD device 100 receives power from a power source throughthe wired cable connected to the connector to charge the battery 230.

A keypad receives a key input from the user for control of the HMDdevice 100. The keypad includes a physical keypad formed in the HMDdevice 100, a virtual keypad displayed by the first projector 270 and/orthe second projector 275, or a combination thereof.

The sensor unit 225 includes at least one sensor for detecting a stateor a surrounding environment state of the HMD device 100. The sensorunit 225 includes a proximity sensor for detecting the user's proximityto the HMD device 100, a motion/orientation sensor for detecting amotion (e.g., rotation, acceleration, deceleration, and vibration) ofthe HMD device 100, an illumination sensor for detecting ambientillumination, or a combination thereof. The motion/orientation sensorincludes at least one of an acceleration sensor, a gravity sensor, ageomagnetic sensor, a gyro sensor, a shock sensor, a Global PositioningSystem (GPS) module, and a compass sensor. The sensor unit 225 detects astate of the HMD device 100 and transmits a signal indicating the stateof the HMD device 100 to the controller 290. The GPS module receiveselectric waves from a plurality of GPS satellites (not illustrated)around the Earth's orbit and calculates the position of the HMD device100 by using a time of arrival of the electric waves from the GPSsatellite (not illustrated) to the HMD device 100, for example. Thecompass sensor calculates a posture or orientation of the HMD device100.

The power manager 235 supplies power to the HMD device 100 under controlof the controller 290. The power manager 235 may be connected to one ormore batteries 230. The power manager 235 may also supply power, whichis received from an external power source (not illustrated) through thewired cable connected with the connector, to the HMD device 100.

The communication unit 240 is a wired or wireless communication devicein a wired or wireless manner, which transmits data from the controller290 or receives data from an external communication line or over the airand delivers the data to the controller 290 in a wired or wirelessmanner.

For example, the communication unit 240 includes at least one of amobile communication module, a wireless Local Area Network (WLAN)module, and a short-range communication module, depending on itsfunctionality.

The mobile communication module enables the HMD device 100 tocommunicate with an electronic device through a mobile communicationnetwork by using one or more antennas (not illustrated) under control ofthe controller 290. The mobile communication module transmits/receives aradio signal for voice communication, video communication, a ShortMessaging Service (SMS), or a Multimedia Messaging Service (MMS) with acellular phone (not illustrated), a smart phone (not illustrated), atablet Personal Computer (PC) (not illustrated), or anothercommunication device having a network address, such as an InternetProtocol (IP), or a phone number.

The WLAN module may be connected to the Internet under control of thecontroller 290 in a place where a wireless Access Point (AP, notillustrated) is installed. The WLAN module supports a WLAN standard(IEEE802.11x) of the Institute of Electrical and Electronics Engineers(IEEE).

The short-range communication module wirelessly performs short-rangecommunication with an external short-range communication device undercontrol of the controller 290. The short-range communication includesBluetooth®, Infrared Data Association (IrDA), WiFi-Direct communication,Near Field Communication (NFC), or a combination thereof.

The touch sensor 250 transmits a signal corresponding to at least onetouch input to the controller 290. The user touches the touch sensor 250by using a finger, for example, or a touch input device such as astylus, and the touch sensor 250 receives a user's touch input. Thetouch sensor 250 receives an input corresponding to continuous movementof a touch (such as a drag input). Touch input information includestouch coordinates and/or a touch state. The touch state includes a mousedown state for pressing the touch sensor 250, a mouse up state forremoving a finger from the touch sensor 250, and a drag state forsliding while pressing the touch sensor 250. The controller 290recognizes selection or movement of a menu item or an icon, or userinput information such as a handwriting input from the touch inputinformation, and performs a function (e.g., phone connection, cameraphotographing, message generation/view, and data transmission)corresponding to the user input information.

Herein, the touch includes a contactless (i.e., non-contact) touch, suchas the touch sensor 250 and a touch input device being spaced apart fromeach other, as well as a contact between the touch sensor 250 and thetouch input device. Such a contactless touch input may also be referredto as a hovering input. The touch sensor 250 may be implemented as aresistive type touch panel, a capacitive type touch panel, an infraredtype touch panel, an acoustic wave type touch panel, an ElectromagneticResonance (EMR) type touch panel, or a combination thereof.

The camera 260 includes a lens system and an image sensor, and mayfurther include a flash. The camera 260 converts an optical signal input(or captured) through the lens systems into an electric image signal andoutputs the electric image signal to the controller 290. The usercaptures a moving image or a still image through the camera 260. Thecamera 260 may also be provided to receive a user input corresponding toa user's motion or gesture.

The lens system forms an image of an object by converging light inputfrom outside. The lens system includes at least one lens, which may beconvex or aspheric. The lens system is symmetrical with respect to anoptical axis that passes through a center thereof, and the optical axisis defined as a central axis. The image sensor detects an optical imageformed by external light that is input through the lens system as anelectric image signal. The image sensor includes a plurality of pixelunits arranged in an MxN matrix, and each pixel unit includes aphotodiode and at least one transistor. The pixel unit accumulates anelectric charge generated by input light, and a voltage based on theaccumulated electric charge indicates an illuminance of the incidentlight. When an image forming a still image or moving image is processed,image data output from the image sensor includes a set of voltages(i.e., pixel values) output from the pixel units, and the image dataindicates one image (i.e., a still image) that includes MxN pixels. Theimage sensor is, e.g., a Charge-Coupled Device (CCD) image sensor or aComplementary Metal-Oxide Semiconductor (CMOS) image sensor.

The image sensor may operate all pixels thereof or only pixels of aRegion Of Interest (ROI) according to a control signal received from thecontroller 290, and outputs image data output from the pixels to thecontroller 290.

The controller 290 processes a Graphic User Interface (GUI) configuredby the controller 290 using an image input from the camera 260, an imagestored in the memory 220, or data stored in the memory 220, in frameunits, and outputs an image frame converted to fit for screen outputcharacteristics (e.g., size, quality, and resolution) of the firstprojector 270 and/or the second projector 275 to the outside through thefirst projector 270 and/or the second projector 275, or stores theconverted image frame in the memory 220. Herein, a GUI is an example ofa virtual image formed by the first projector 270 and/or the secondprojector 275, but the expression “a virtual image” may also be usedinstead of the GUI and the virtual image includes a virtual object thatis not a real object, such as a GUI or still image content.

The controller 290 provides a GUI corresponding to various services(e.g., phone conversation, data transmission, broadcasting, andphotographing) to the user through the first projector 270 and/or thesecond projector 275. The controller 290 provides a still image or amoving image to the user through the GUI. That is, in the presentinvention, the GUI may illustrate a screen expressed with a still imageor a moving image.

The first projector 270 and the second projector 275 have the samestructure, and each projects light that forms the virtual image providedby the controller 290 to the user's eyes through the first window 280and the second window 285, respectively.

FIG. 4 illustrates a schematic structure of the first projector 270.

The first projector 270 includes a light source 410 for outputtinglight, an illumination optical system 420 for illuminating the displaydevice 440 with light output from the light source 410, a mirror 430 forreflecting light that passes through the illumination optical system420, the display device 440 for reflecting the light reflected by themirror 430 in pixel units to form a virtual image, and a projectionoptical system 450 for projecting the light reflected from the displaydevice 440 to the outside.

The illumination optical system 420 has a first optical axis 271 that isparallel with an X-axis, and includes at least one collimation lens, atleast one filter, at least one equalization lens, a condensing lens, ora combination thereof.

Optical elements such as a lens, prism, and filter of the illuminationoptical system 420 are aligned with the first optical axis 271.Generally, an optical axis does not experience an optical change, evenwhen a corresponding optical system rotates around the axis. Alignmentwith an optical axis indicates that a curvature center of an opticalelement of the optical system is positioned on the optical axis or asymmetric point (i.e., a symmetric center) or a center point of theoptical element is positioned on the optical axis.

The light source 410 outputs light that travels along the first opticalaxis 271. At least one Light Emitting Diode (LED) for outputting whitelight, primary light (e.g., blue light or green light), or a combinationof primary lights is used as the light source 410.

The illumination optical system 420 collimates, filters, and/orcondenses light input from the light source 410, and outputs theprocessed light to the mirror 430.

The mirror 430 reflects the input light passing through the illuminationoptical system 420 toward the display device 440. A high-reflectivitydielectric layer or metallic layer may be deposited on a substrate ofthe mirror 430.

The display device 440 displays an image in pixel units according todata input from the controller 290, and the display device 440 includespixel elements corresponding to a preset resolution and displays animage through on/off of the pixel elements. A Digital Micro-MirrorDevice (DMD) including micro mirrors arranged in an MxN (e.g., 1280x720or 854x480) matrix structure, for example, may be used as the displaydevice 440. Each micro mirror rotates to a position corresponding to anon or off state according to a drive signal, and reflects incident lightincident at an angle that allows a display to the outside in the onstate and reflects incident light at an angle that does not allow adisplay to the outside in the off state.

The projection optical system 450 has a second optical axis 272 that isparallel with a Z-axis, and includes a relay lens 460 and a projectionlens 470 which are both aligned with the second optical axis 272.

The relay lens 460 causes light reflected from the mirror 430 to bematched to the display device 440, considering overfill. That is, therelay lens 460 causes the light reflected from the mirror 430 to beincident to an area that is equal to or larger than an area occupied bypixel devices of the display device 440.

The relay lens 460 also receives the light reflected from the displaydevice 440, and outputs the light after reducing the beam spot size ofthe light. The light reflected from the display device 440 has a largebeam spot size, and thus a large light loss may occur due to light thatfails to be delivered to the projection lens 470. Therefore, the relaylens 460 collects the light reflected from the display device 440 andreduces the beam spot size of the light, thereby delivering as muchlight as possible to the projection lens 470.

The projection lens 470 receives light having a beam spot size that isadjusted from the relay lens 460, collimates or condenses the receivedlight, and projects the light to the outside.

Referring back to FIG. 3, the first window 280 and the second window 285have the same structure, and each has a transmissivity that varies undercontrol of the controller 290 and has a hologram pattern which functionsas a concave mirror.

FIG. 5 illustrates a structure and a function of the first window 280according to an embodiment of the present invention.

The first window 280 includes first glass 281 having a transmissivitythat varies according to an applied signal, and a first HolographicOptical Element (HOE) 282 which functions as a concave mirror. Likewise,the second window 285 includes second glass having a transmissivity thatis controlled and a second HOE which functions as a concave mirror.

The first window 280 passes therethrough ambient light 511 input fromthe outside, and reflects and condenses first projection light 521 inputfrom the first projector 270.

The ambient light 512 that passes through the first window 280 and thefirst projection light 522 reflected by the first window 280 are inputto the left eye 330. An ambient landscape image 510 formed by theambient light 512 and a GUI 520 formed by the reflected first projectionlight 522 are formed in an overlapping manner on a retina 333 of theleft eye 330. That is, the user sees an image formed by overlapping ofthe ambient landscape image 510 and the GUI 520, and to the user, theGUI 520 may be seen as if a transparent layer (i.e., a GUI) overlaps anambient landscape. The GUI 520 includes menu items (or icons) such as“Phone”, “Contacts”, “Message”, and “Applications”. In FIG. 5, the GUI520 is displayed opaque, but the GUI 520 may be displayed partially orentirely transparent to allow the ambient landscape image 510 under theGUI 520 to be shown through the GUI 520.

The first projection light 521 output from the first projector 270 isparallel light (that is, collimated light) having a particularwavelength λ, and the first projection light 521 is incident to thefirst HOE 282 while forming a particular angle θ with a normal line ofthe first HOE 282. The first HOE 282 is an element having wavelengthselectivity, such that the first HOE 282 reflects and condenses lighthaving the particular wavelength λ (i.e., the first projection light521) and passes therethrough light having wavelength other than theparticular wavelength λ (i.e., the ambient light 511) without convergingthe light.

The first HOE 282 reflects and condenses the input first projectionlight 521, and the first projection light 522 reflected from the firstHOE 282 converges to the left eye 330 spaced apart from the first HOE282 by a particular distance, that is, an eye relief, preferably to apupil 331 or a crystalline lens 332 of the left eye 330. The convergingfirst projection light 522 has a particular convergence angle or viewingangle φ. The crystalline lens 332 adjusts a focus of light incident tothe eye, and the passing ambient light 512 converges into an eyeball 334by the crystalline lens 332, thus forming the ambient landscape image510 on the pupil 333. The reflected first projection light 522 convergesto the pupil 331 or the crystalline lens 332, such that the firstprojection light 522 is projected on the pupil 333 without converging bythe crystalline lens 332, thus forming the GUI 520.

Although the first projection light 521 incident to the first HOE 282 isillustrated as being parallel light, the present invention is notlimited thereto, such that the first projection light 522 reflected bythe first HOE 282 may be light or convergence light instead of parallellight, such that the reflected first projection light 522 converges tothe pupil 331 or the crystalline lens 332.

If the reflected first projection light 522 is converged by thecrystalline lens 332, a virtual object (or an object) of a virtual imagesuch as a GUI may not be clearly imaged on the retina of a user when theuser who wears the HMD device 100 adjusts the focus of the eyes to see areal object of an ambient landscape. In this situation, the user failsto clearly see the real object and the virtual object concurrently, thusfailing to recognize intended augmented reality.

The HMD device 100 according to an embodiment of the present inventionconverges projection light for forming a virtual image to a pupil or acrystalline lens, thus solving a conventional problem. That is, the HMDdevice 100 causes the projection light to converge to or close to thecrystalline lens, thereby minimizing a focus change of the projectionlight, caused by the crystalline lens.

As such, when the projection light converges to the pupil or thecrystalline lens, the projection light is directly projected to theretina regardless of focus adjustment of the crystalline lens of theeye. A virtual image projected on the retina is recognized by the useras a clear image, irrespective of focus adjustment and aberration of thecrystalline lens.

The first HOE 282 may have a focal length corresponding to a distancebetween the pupil 331 or the crystalline lens 332 and the first HOE 282,and in this case, the virtual image is clearly formed on the retina ofthe eye regardless of focus adjustment of the eye

FIG. 6 illustrates a structure of the first glass 281 according to anembodiment of the present invention. The first glass 281 has atransmissivity that varies according to a signal or voltage applied bythe controller 290.

The first glass 281 may use, as examples, electrochromic glass, aSuspended Particle Device (SPD), or Liquid Crystal (LC). In some cases,the first glass 281 may use photochromic glass or thermochromic glassthat may not be actively controlled by an applied signal, and that has atransmissivity changing in reaction with light of a particularwavelength or a temperature change.

The first glass 281 is manufactured in various manners, such as byapplying a transmissivity-adjustable material onto glass or by attachinga transmissivity-adjustable thin film onto glass.

In the current example, the first glass 281 uses electrochromic glass.

The first glass 281 includes an insulating first substrate 610 and aninsulating second substrate 615, a conductive first electrode 620stacked on a top surface of the first substrate 610, a conductive secondelectrode 625 stacked on a bottom surface of the second substrate 615,an insulating spacer 630 for separating the first substrate 610 from thesecond substrate 615 and sealing a space between the first substrate 610and the second substrate 615, and an electrochromic layer 640 and anelectrolyte 650 that are filled in the space between the first substrate610 and the second substrate 615.

Each of the first substrate 610 and the second substrate 615 is made oftransparent glass or plastic, and the plastic is one of polyacrylate,polyethylene etherphthalate, polyethylene naphthalate, polycarbonate,polyarylate, polyether imide, polyether sulfone, and polyimide, forexample.

The first electrode 620 is made of a transparent conductor, andincludes, e.g., an inorganic conductive material such as Indium TinOxide (ITO), Fluorine Tin Oxide (FTO), or Antimony Doped Tin Oxide(ATO), or an organic conductive material such as polyacetylene orpolythiophene.

The second electrode 625 is made of a transparent or an opaqueconductive material, and includes, e.g., ITO, FTO, metal such as Al,ATO, or a combination thereof

The electrochromic layer 640 including an electrochromic material isdisposed on the first electrode 620, in the form of a film.

The first substrate 610 and the second substrate 615 are fixed by thespacer 630, and the electrolyte 650 is filled between the firstsubstrate 610 and the second substrate 615. The electrolyte 650 providesan oxidation/reduction material that reacts with an electrochromicmaterial, and is a liquid electrolyte or a solid high-polymerelectrolyte. The liquid electrolyte may use, for example, a solutionmade by dissolving lithium salt such as LiOH or LiClO4, potassium saltsuch as KOH, and sodium salt such as NaOH in a solvent. The solidelectrolyte uses, for example, poly(2-acrylamino-2-methylpropanesulfonic acid) or poly(ethylene oxide)).

The material of the electrochromic layer 640, that is, theelectrochromic material includes a metal-organic complex in which metaland an organic compound having a functional group capable of formingcoordination with the metal are combined. The metal includes lightmetal, transition metal, lanthanide metal, alkali metal, or acombination thereof, and the metal includes Beryllium (Be), Barium (Ba),Copper (Cu), Zinc (Zn), Cerium (Ce), Magnesium (Mg), Aluminum (Al),Titanium (Ti), or a combination thereof. The functional group includes acarboxyl group, a pyridine group, an imidazole group, or a combinationthereof. The organic compound includes a viologen derivative, ananthraquinone derivative, or a combination thereof.

FIGS. 7A and 7B illustrate holographic patterns 710 and 710 a of firstHOEs 282 and 282 a according to an embodiment of the present invention.

The first HOEs 282 and 282 a include the holographic patterns 710 and710 a including multiple concentric circles. The first HOEs 282 and 282a includes, for example, a transparent substrate and a holographicpattern layer stacked on the transparent substrate.

FIG. 7A illustrates an example of the first HOE 282 in which the centerof the concentric circles is in the center of the holographic pattern710, and FIG. 7B illustrates an example of the first HOE 282 a in whichthe center of the concentric circles is in an edge of the holographicpattern 710 a.

Referring back to FIG. 3, the controller 290 controls the overalloperation of the HMD device 100, and controls other components in theHMD device 100 to provide a method for providing a virtual image. Thecontroller 290 includes a single-core, dual-core, triple-core, orquad-core processor. The controller 290 receives a broadcast signal(e.g., a TeleVision (TV) broadcast signal, a radio broadcast signal, ora data broadcast signal) and broadcast additional information (e.g., anElectronic Program Guide (EPG) or an Electronic Service Guide (ESG))transmitted from a broadcasting station through the communication unit240.

The controller 290 plays a digital audio file (e.g., a file having afile extension such as ‘mp3’, ‘wma’, ‘ogg’, or ‘way’) stored in thememory 220 or received through the communication unit 240 through thespeaker 211. The controller 290 plays a digital video file (e.g., a filehaving a file extension such as ‘mpeg’, ‘mpg’, ‘mp4’, ‘avi’, ‘mov’, or‘mkv’ stored in the memory 220 or received through the communicationunit 240 through the first projector 270 and/or the second projector275. The controller 290 displays image data (such as a GUI) configuredby the controller 290 to the user through the first projector 270 and/orthe second projector 275 by using data stored in the memory 220 orreceived through the communication unit 240 according to a user command,a selection of a menu item or an icon, or event information inputthrough the sensor unit 225, the input/output module 210, the camera260, or the touch sensor 250. The image is a still or moving image.

The memory 220 stores a signal or data under control of the controller290. The memory 220 stores a control program and applications for theHMD device 100 or the controller 290.

Herein, the term “memory” includes a Read Only Memory (ROM) or a RandomAccess Memory (RAM) in the controller 290 or a memory card (not shown)(e.g., a Secure Digital (SD) card or a memory stick), a non-volatilememory, a volatile memory, or a Solid State Drive (SSD) mounted on theHMD device 100.

FIGS. 8A through 9B describe transmittance control of the HMD device100. The first window 280 and the second window 285 controltransmissivities under the control of the controller 290, thus improvingvisibility of the virtual image.

Since the first window 280 and the second window 285 adjusttransmissivities according to a change of an applied voltage, output ofthe first projector 270 and/or the second projector 275 for forming avirtual image is reduced, thus reducing total power consumption and heatemission of the first projector 270 and/or the second projector 275 andincreasing the lifespan of the battery 230 of the HMD device 100.

FIG. 8A and 8B illustrate when the user watches a TV in an indoorenvironment.

In an indoor environment having a low ambient illumination, visibilityof the first GUI 820 and visibility of the second GUI 825 formed by thefirst projector 270 and the second projector 275 are high, and thustransmissivities of the first window 280 and the second window 285 areset relatively high. The controller 290 sets the transmissivities of thefirst window 280 and the second window 285 to maximal values or to 30%or higher, for example. The first GUI 820 and the second GUI 825 are thesame as each other except that they are displayed in differentpositions.

FIGS. 9A and 9B illustrate when the user views an ambient landscape inan outdoor environment.

In an outdoor environment having a high ambient illumination, visibilityof the first GUI 920 and visibility of the second GUI 925 formed by thefirst projector 270 and the second projector 275 are low, such that thetransmissivities of the first window 280 and the second window 285 areset relatively low. The controller 290 sets the transmissivities of thefirst window 280 and the second window 285 to minimal values or to 10%or lower, for example.

In FIGS. 8A through 9B, the first GUIs 820 and 920 and the second GUIs825 and 925 are not images that are formed on the first window 280 andthe second window 285, but are images shown to the user. In FIGS. 8Athrough 9B, each GUI is displayed opaque, but the GUI may be displayedpartially or entirely transparent to allow the ambient landscape underthe GUI to be shown through the GUI.

The controller 290 measures an ambient illumination through the sensorunit 225, increases the transmissivities of the first window 280 and thesecond window 285 if the ambient illumination is lower than a referenceillumination or reference illumination range (i.e., the first window 280and the second window 285 have relatively high transmissivities), andreduces the transmissivities of the first window 280 and the secondwindow 285 if the ambient illumination is higher than the referenceillumination or reference illumination range (i.e., the first window 280and the second window 285 have relatively low transmissivities). Thereference illumination may be a currently set ambient illumination. Thecontroller 290 may maintain the transmissivities of the first window 280and the second window 285 if there is no change in the ambientillumination. The controller 290 stores the currently set ambientillumination and/or transmissitivies in the memory 220.

The memory 220 stores a data table indicating ambient illuminationvalues and transmissitivies (and/or applied voltage values of the firstwindow 280 and the second window 285) that correspond to each other. Thecontroller 290 calculates a target transmissivity (and/or an appliedvoltage value of each of the first window 280 and the second window 285)based on the data table through mapping, interpolation or equationcalculation. The controller 290 applies a voltage corresponding to thecalculated transmissivity to each of the first window 280 and the secondwindow 285, which are glass, thereby adjusting the transmissitivies ofthe first window 280 and the second window 285 to the targettransmissivity.

FIG. 10 illustrates a method for providing a virtual image according toan embodiment of the present invention.

In FIG. 10, a GUI is illustrated as an example of the virtual image.

In step S1010, the controller 290 measures an ambient illumination byusing the sensor unit 225 or the camera 260. The sensor unit 225includes an illumination sensor, and an ambient illumination valuemeasured by the sensor unit 225 is output to the controller 290 from thesensor unit 225. The camera 260 converts light that forms an ambientlandscape input (or captured) through a lens system into an electricimage signal and outputs the electric image signal to the controller290, such that the controller 290 may measure the ambient illuminationby using a brightness of the light.

In step 1020, the controller 290 calculates a target transmissitivity(and/or an applied voltage value of each of the first window 280 and thesecond window 285) corresponding to the ambient illumination value, byusing a data table including ambient illumination values andtransmissivities that are stored in the memory 220 corresponding to eachother.

In step S1030, the controller 290 applies a voltage corresponding to thecalculated target transmissivity to glass of each of the first window280 and the second window 285, thus adjusting the transmissitivies ofthe first window 280 and the second window 285 to the calculated targettransmissivity. That is, the controller 290 controls the first window280 and the second window 285 such that the transmissivities of thefirst window 280 and the second window 285 are equal to the calculatedtarget transmissivity.

In step S1040, the controller 290 configures a first GUI by using datastored in the memory 220 and displays the configured first GUI to theuser through the first projector 270 and/or the second projector 275.The first GUI may be a basic GUI that is initially displayed to the userwhen the HMD device 100 is powered on or starts, and e.g., the GUI 520illustrated in FIG. 5 may be displayed to the user.

In step S1050, the controller 290 receives a user input through theinput/output module 210, the touch sensor 250, the camera 260, or thecommunication unit 240. The user selects the button 213 or an icon or amenu item, input a voice command through the microphone 212, performs agesture or a motion input through the camera 260, or wirelessly inputs aparticular command through the communication unit 240. The command is anexecution command for an application, which for example is an arbitraryapplication such as a voice recognition application, a schedulemanagement application, a document generation application, a musicapplication, an Internet application, a map application, a cameraapplication, an e-mail application, an image editing application, asearch application, a file explorer application, a video application, agame application, a Social Networking Services (SNS) application, aphone application, or a message application. The gesture or motion inputindicates that the user draws a trajectory of a pattern such as acircle, a triangle, or a rectangle toward the camera 260 with a hand ora finger. Although an application is executed according to a user inputin this example, the application may also be automatically executed uponoccurrence of an event such as message reception, call reception, or analarm event.

In step S1060, the controller 290 configures a second GUI by using datastored in the memory 220 according to a user input and displays theconfigured second GUI to the user through the first projector 270 and/orthe second projector 275. The second GUI may be an application window.Changing the first GUI into the second GUI may be described as updatingthe GUI according to the user input.

In the following description of examples of the method for providing avirtual image, only the GUI displayed by the first projector 270 and thefirst window 280 is illustrated, but such a description may be equallyapplied to the second GUI displayed by the second projector 275 and thesecond window 285.

FIGS. 11 through 13 describe a first example of the method for providinga virtual image according to an embodiment of the present invention.

Referring to FIG. 11, the user selects the button 213, an icon or a menuitem, inputs a voice command, a gesture, a motion, or a touch pattern,and the controller 290 executes a voice recognition applicationcorresponding to the user input. The controller 290 configures anapplication window 1110 a by using data stored in the memory 220 anddisplays the configured application window 1110 a to the user throughthe first projector 270 and the first window 280.

In the following examples illustrated in FIGS. 11 through 13, the voicerecognition application window 1110 a is shown to the user, and is notdisplayed on the first window 280.

While the voice recognition application and another application areillustrated as the subjects of program operations, that the controller290 may also perform the program operations.

FIG. 11 illustrates an initial screen of the voice recognitionapplication.

Once the voice recognition application is initially driven, a use guidephrase 1130 such as “What would you like to do?” is displayed on theapplication window 1110 a.

A voice recognition button 1120 for executing a voice recognition modeis disposed in a lower portion of the voice recognition applicationwindow 1110 a. A voice guide button for guiding a using method withvoice may be disposed in a side of the voice recognition button 1120. Ahelp button for displaying examples of the using method may be disposedin the other side of the voice recognition button 120.

Referring to FIG. 12, the user inputs a desired command, such as“Weather in Seoul” in the current example, by voice through themicrophone 212.

The voice recognition application recognizes a voice command input bythe user 10 and converts the voice command into text data 1140.

The voice recognition application displays the text data 140 on anapplication window 1110 b.

Referring to FIG. 13, the voice recognition application searches forweather in Seoul by using the text data 1140 as a search word, anddisplays search results 1150 and 1160, that is, a guide phrase 1150 andweather information 1160, on an application window 1110 c. The voicerecognition application may search for weather in Seoul by using thetext data 1140 and a current location (e.g., Seokyo-dong) of the HMDdevice 100 as search words.

The voice recognition application converts the text data 1140 into voicedata, transmits the voice data to a voice recognition server, andprovides a response result received from the voice recognition server tothe user. Alternatively, the voice recognition application transmits thetext data 1140 to the voice recognition server and provides a responseresult received from the voice recognition server to the user.

FIG. 14 illustrates a second example of the method for providing avirtual image according to an embodiment of the present invention.

Referring to FIG. 14, the user selects a button 213, an icon or a menuitem, or inputs a voice command, a gesture, motion, or a touch patternthrough the input/output module 210 or the camera 260, and thecontroller 290 executes a music application corresponding to the userinput. The controller 290 configures a music application window 1410 byusing data stored in the memory 220, and displays the music applicationwindow 1410 to the user through the first projector 270 and the firstwindow 280.

The music application plays a music file according to a user's selectionor a music file that is set by default, and displays a title and aplaytime of a currently played music file on the music applicationwindow 1410. In a lower portion of the music application window 1410, amenu item 1420 such as pause, fast-forward, and rewind and a listsbutton for displaying a selectable music list may be provided.

FIG. 15 illustrates a third example of the method for providing avirtual image according to an embodiment of the present invention.

Referring to FIG. 15, the user selects the button 213, an icon or a menuitem, or inputs a voice command, a gesture, a motion, or a touch patternthrough the touch sensor 250, the input/output module 210 or the camera260, and the controller 290 executes a call application corresponding tothe user input. The controller 290 configures a call application window1510 by using data stored in the memory 220, and displays the configuredcall application window 1510 to the user through the first projector 270and the first window 280.

The call application displays a keypad 1520 for inputting a phone numberand menu items 1530 such as keypad conversion, recent logs, contacts,and favorites on the call application window 1510.

FIG. 16 illustrates a fourth example of the method for providing avirtual image according to an embodiment of the present invention.

Referring to FIG. 16, the user selects the button 213, an icon or a menuitem, or inputs a voice command, a gesture, a motion, or a touch patternthrough the touch sensor 250, the input/output module 210 or the camera260, and the controller 290 executes a camera application correspondingto the user input. The controller 290 configures a camera applicationwindow 1610 by using data stored in the memory 220, and displays theconfigured camera application window 1610 to the user through the firstprojector 270 and the first window 280.

The camera application displays a photographing button 1620 forphotographing, a photographing position or focus position indicator1630, and menu items 1640 such as environment setting and a flash, onthe camera application window 1610.

FIG. 17 illustrates a fifth example of the method for providing avirtual image according to an embodiment of the present invention.

Referring to FIG. 17, the user selects the button 213, an icon or a menuitem, or inputs a voice command, a gesture, a motion, or a touch patternthrough the touch sensor 250, the input/output module 210 or the camera260, and the controller 290 executes a message application correspondingto the user input. The controller 290 configures a message applicationwindow 1710 by using data stored in the memory 220, and displays theconfigured message application window 1710 to the user through the firstprojector 270 and the first window 280.

The message application displays a keypad 1720 for text input and menuitems 1730 such as message transmission and file attachment, on themessage application window 1710.

According to an embodiment of the present invention, a virtual image anda real image are concurrently provided by using an HOE-based coupler,window, and reduces weight and size of the HMD device by integrallyforming glass and a HOE. In addition, a window adjusts a transmissivityof light with an electric signal, thus improving outdoor visibility ofthe virtual image, The window is driven with low power, thus reducingheat emission and lengthening the battery lifespan. Moreover, the windowhas a large Field of View (FoV) and good display quality.

The above-described embodiments of the present invention may beimplemented with hardware, software, or a combination of hardware andsoftware. The software may be stored in a volatile or non-volatilestorage such as a Read-Only Memory (ROM), a memory such as a RandomAccess Memory (RAM), a memory chip, a device, or an integrated circuit,and an optically or magnetically recordable and machine (e.g.,computer)-readable storage medium such as a Compact Disc (CD), a DigitalVersatile Disk (DVD), a magnetic disk, or a magnetic tape. A memorywhich can be included in the HMD device includes, e.g., amachine-readable storage medium which is suitable for storing a programor programs including instructions for implementing the audio contentplayback method according to the embodiment of the present invention.Therefore, the present invention includes a program including codes forimplementing the audio content playback apparatus or method according tothe embodiments of the present invention and a machine-readable storagemedium for storing such a program. The program may be electronicallytransferred through a medium such as a communication signal deliveredthrough wired or wireless connection, and the present invention properlyincludes equivalents thereof.

The HMD device receives and stores the program from a program providingdevice connected in a wired or wireless manner. The program providingdevice includes a memory for storing a program including instructionsfor instructing the HMD device to execute a preset operating mode,information necessary for the operating mode, a communication unit forperforming wired or wireless communication with the electronic paper,and a controller for transmitting a corresponding program to the HMDdevice at the request of the HMD device or automatically.

While the present invention has been particularly illustrated anddescribed with reference to certain embodiments thereof, variousmodifications or changes can be made without departing from the scope ofthe present invention. Therefore, the scope of the present invention isnot limited to the described embodiments, should be defined by the scopeof the following claims and any equivalents thereof.

What is claimed is:
 1. A method for providing a virtual image to a userin a Head-Mounted Display (HMD) device, the method comprising: detectingan ambient illumination; calculating a target transmissivity of a windowprovided in the HMD device, based on the ambient illumination; adjustinga transmissivity of the window based on the calculated targettransmissivity; and providing a virtual image to the user by projectinglight to the window from a projector provided in the HMD device.
 2. Themethod of claim 1, wherein the projected light reflected by the windowconverges to a pupil or crystalline lens of an eye of the user.
 3. Themethod of claim 1, wherein the transmissivity of the windowcorresponding to the detected ambient illumination is calculated byusing a data table that indicates ambient illumination values andcorresponding transmissivities of the window.
 4. The method of claim 1,wherein the transmissivity of the window is reduced when the detectedambient illumination is higher than a currently set ambientillumination, and the transmissivity of the window is increased when thedetected ambient illumination is lower than the currently set ambientillumination.
 5. The method of claim 1, further comprising: receiving aninput of the user; and updating and displaying the virtual imageaccording to the input of the user.
 6. The method of claim 1, whereinthe virtual image is a Graphic User Interface (GUI) corresponding to auser input or event occurrence.
 7. A non-transitory computer-readablestorage medium having recorded thereon a program for executing a methodfor providing a virtual image to a user in a Head-Mounted Display (HMD)device, the method comprising: detecting an ambient illumination;calculating a target transmissivity of a window provided in the HMDdevice, based on the ambient illumination; adjusting a transmissivity ofthe window based on the calculated target transmissivity; and providinga virtual image to the user by projecting light to the window from aprojector provided in the HMD device.
 8. A Head-Mounted Display (HMD)device comprising a computer-readable storage medium having recordedthereon a program for executing a method for providing a virtual imageto a user in a Head-Mounted Display (HMD) device, the method comprising:detecting an ambient illumination; calculating a target transmissivityof a window provided in the HMD device, based on the ambientillumination; adjusting a transmissivity of the window based on thecalculated target transmissivity; and providing a virtual image to theuser by projecting light to the window from a projector provided in theHMD device.
 9. A Head-Mounted Display (HMD) device for providing avirtual image to a user, the HMD device comprising: a sensor unitconfigured to detect an ambient illumination; a projector configured toproject light; a window configured to condense and reflect the lightprojected by the projector and to provide a virtual image formed by thereflected light to the user; and a controller configured to calculate atarget transmissivity of the window corresponding to the detectedambient illumination and adjust a transmissivity of the window accordingto the target transmissivity.
 10. The HMD device of claim 9, wherein thewindow passes ambient light therethrough.
 11. The HMD device of claim 9,wherein the light reflected by the window converges to a pupil orcrystalline lens of an eye of the user.
 12. The HMD device of claim 9,wherein the window comprises: a glass having a transmissivity thatchanges with a signal applied from the controller; and a HolographicOptical Element (HOE) having a holographic pattern for reflecting andcondensing the light input from the projector.
 13. The HMD device ofclaim 9, further comprising a memory for storing a data table thatindicates ambient illumination values and corresponding transmissivitiesof the window,, wherein the controller calculates the transmissivity ofthe window corresponding to the detected ambient illumination by usingthe data table.
 14. The HMD device of claim 9, wherein the controllerapplies a voltage corresponding to the target transmissivity to thewindow to adjust the transmissivity of the window.
 15. The HMD device ofclaim 9, wherein the controller reduces the transmissivity of the windowwhen the detected ambient illumination is higher than a currently setambient illumination, and increases the transmissivity of the windowwhen the detected ambient illumination is lower than the currently setambient illumination.